Bachelor of Science in Computer Science 
May 2022 


A comparison through heuristic 
evaluation of two MOOC platforms: 
Coursera and Udemy 


Harshitha Gundala 
Sada Siva Kumar Boya 


Faculty of Computing, Blekinge Institute of Technology, 371 79 Karlskrona, Sweden 


This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in 
partial fulfilment of the requirements for the degree of Bachelor of Science in Computer Science. 
The thesis is equivalent to 10 weeks of full time studies. 


The authors declare that they are the sole authors of this thesis and that they have not used 
any sources other than those listed in the bibliography and identified as references. They further 
declare that they have not submitted this thesis at any other institution to obtain a degree. 


Contact Information: 
Author(s): 

Harshitha Gundala 

E-mail: hagn21@student.bth.se 


Sada Siva Kumar Boya 
E-mail: saby21@student.bth.se 


University advisor: 
Dr. Sai Prashanth Josyula, Bitradande Lektor 
Department of Computer Science 


Faculty of Computing Internet : www.bth.se 
Blekinge Institute of Technology Phone : +46 455 38 50 00 
SE-371 79 Karlskrona, Sweden Fax : +46 455 38 50 57 


Abstract 


In recent years thousands of people worldwide have been using Massive open on- 
line courses (MOOCs) for various reasons, including career development and change, 
continuing education, lifelong learning, and training. MOOCs are free online courses 
that anyone can enroll in and are an affordable and flexible way to learn new skills, 
develop professionally, and provide quality educational experience across the globe. 
Having an exemplary user interface will make users prefer most MOOC platforms, 
which inspired this study to know which MOOC platform has usability and better 
user experience by using the survey evaluation. 


The main objective of this study is to measure the usability components(effectiveness, 
efficiency, and overall user satisfaction level) to determine which MOOC platform, 
Coursera or Udemy, has a better user experience using the heuristic principles. 


A quantitative research method has been used in this paper, which focuses on sta- 
tistical, mathematical, and numerical analysis of data collected through surveys, 
questionnaires, and polls. In this paper, a questionnaire to determine user satis- 
faction through surveys has been developed. Usability has been evaluated for each 
MOOC platform (Coursera and Udemy) by calculating usability components. Sta- 
tistical methods like variance and standard deviations has been calculated using the 
survey results. 


Responses from the survey evaluation are collected and are graphically represented 
in bar graphs and pie charts. Respondent’s timings and statistical method results 
are presented in a tabular form. The satisfaction level is calculated and represented 
in a Likert scale format. 


The usability of two MOOC platforms (Coursera and Udemy) was measured by cal- 
culating usability components. Better user experience among Coursera and Udemy 
is measured using usability heuristic principles. This study concludes that effec- 
tiveness, efficiency, and satisfaction level while performing eight tasks given to the 
user is similar in both MOOC platforms (Coursera and Udemy). From the survey 
evaluation, the users preferred Udemy over Coursera. 


Keywords: Heuristic principles, Massive open online courses (MOOCs), Survey 
evaluation, Usability, User experience. 
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Chapter 1 


Introduction 


Today, one of the most advanced forms of distance education is the massive open 
online course (MOOC), which provides free access and intellectual innovation to stu- 
dents worldwide via the Internet and various technological advances [20]. The reach 
of educational institutions through the MOOC platforms continues to grow as more 
and more students seek to enroll in courses that are easily accessible to anyone in 
the world with an Internet connection [8]. 


Massive open online courses (MOOCs) are a disruptive trend in education. Most 
MOOGCs are offered by dedicated online platforms such as Coursera, Udemy, Edx, 
and LinkedIn Learning, and most attract thousands of learners. However, the effec- 
tiveness of learning in many MOOCs is questionable, as evidenced by low student 
retention rates [18]. 


Studies show that most MOOCs have a completion rate of less than 7% [26]. There 
are several reasons for this lower performance, one of which is related to usability 
issues in terms of user experience. Usability is the ability of a particular user to 
use a system efficiently, effectively, and satisfactorily in a given user environment to 
achieve a particular goal [25]. Usability has a significant impact on student learning 
and the overall learning experience. Therefore, usability is an essential requirement 
for educational software. 


Usability |4] is applied to improve the flexibility and ease of MOOC platforms using 
heuristics evaluation [15]. A survey will be conducted using heuristics principles to 
determine which platforms provide the best user experience. The usability heuristics 
used to survey the MOOC platforms are the following |16] [14] 


1. Visibility of system status 

2. Match between the system and the real world 
3. User control and freedom 

4. Consistency and standards 

5. Error prevention 

6. Recognition rather than recall 

7. Flexibility and efficiency of use 


8. Aesthetics and minimalistic design 


4 Chapter 1. Introduction 


9. Help users recognize, diagnose, and recover from errors 


10. Help and documentation 


These are heuristics that have been used in traditional usability evaluations. Most 
evaluations incorporate these principles to solve usability problems [1] [9]. This study 
will help developers create a better platform for MOOCs or improve the existing 
platforms by following the heuristics and providing a better user experience [12]. 
This study will also help students choose the best platforms to improve their general 
and academic skills. 


1.1 Aim and objectives 


The thesis aims to evaluate user satisfaction based on usability heuristics between 
Coursera and Udemy by surveying long term (users who are using both platforms 
for an extended period) and new users (users who are new to both platforms). 


1.1.1 Objectives 


e To identify the heuristic problems on selected MOOC platforms based on a 
questionnaire survey. 


e ‘To measure the chosen MOOC platforms’ effectiveness, efficiency, and satisfac- 
tion level. 


e To calculate and compare the outcomes of the above platforms using statistical 
methods. 


1.2 Research Questions 


1. How effective, efficient and satisfactory are the two MOOC platforms (Udemy 
and Coursera)? 


Motivation: According to the ISO 9241-11 standard, usability is defined as 
"The extent to which a product can be used by specified users to achieve spec- 
ified goals with effectiveness, efficiency, and satisfaction in a specified context 
of use" [3]. Therefore, usability consists of three significant aspects: effective- 
ness, efficiency, and satisfaction. Therefore, the primary motivation behind 
this research question is to find which MOOC platform is better in usability. 
This RQ can be achieved by assigning tasks to the users and calculating the 
effectiveness, efficiency and satisfaction level using statistical methods. 


2. How does the user experience of two major MOOC platforms (Coursera and 
Udemy) compare to each other? 


Motivation: With the popularity of these MOOC platforms (Coursera and 
Udemy) and the increase in the number of courses, more universities are en- 
gaging in this form of online teaching and learning. At the same time, various 
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course choices create a larger pool of participants who are determined by their 
curiosity and interest in the course context. The platform’s user experience 
can also be linked to increased student enrollment. This is the motivation for 
comparing the user experiences of both platforms, which will give the user an 
idea about which platform to choose. This can be achieved by conducting a 
questionnaires survey among the long-term users and new users of the specified 
MOOC platforms by using usability heuristic principles. 


1.3. Background 


1.3.1 Definition of Usability 


Usability is a quality characteristic used to evaluate the ease of use of a user interface. 
The term "usability" also refers to improving usability during the design process [17]. 


Five quality factors that define usability are 


Learnability: the ease with which users can perform basic tasks when new to the 
design. 


Efficiency: How quickly users can complete tasks after learning the design. 


Memorability: The ease with which users can improve their skills when re-familiarizing 
themselves with the design after not using it for a while. 


Errors: How many mistakes the user makes, how serious they are, and how easily 
they can be corrected. 


Satisfaction: To know the user’s expectations or needs in any design. 


1.3.2 Heuristic Principles 


Heuristic evaluation is a usability method that examines a user interface to identify 
usability problems. Jakob Neilsen developed ten heuristics |16|to identify usability 
problems in user interfaces. These usability heuristics principles are described below. 
1. Visibility of system status 

Users need to be informed of what is happening in the system in a timely manner 
and through appropriate feedback. This approach allows the user to know the state 
of the system and the results of each interaction. The system should indicate the 
status at each stage of the interaction and communicate with the user to provide 
appropriate feedback. 

2. Match between system and the real world 

Interactive systems must communicate using words, phrases, and concepts known to 
the user rather than a system-oriented language. Information must follow real-world 
norms to appear logical and natural. 

3. User control and freedom 

Users can make mistakes when interacting with the system interface. Interactive 
systems must inform users that they should refrain from unnecessary actions that 
could delay the process. Systems must give users the ability to block or interrupt 
any program to give them flexibility and confidence. The exit point of an interactive 
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system allows the user to manipulate the system. 

4. Consistency and standards 

The system must not confuse users by using the same meaning in different phrases 
and situations. The system must comply with specific regulations and industry con- 
ventions. Consumers will spend less time on your digital products and be attracted 
to new products. Therefore, the mental pressure of users learning something new 
may increase due to consistency. 

5. Error prevention 

Error reporting is essential, but the best user interface design is about solving prob- 
lems, not about what happens next. An interactive system should exclude or identify 
error-prone situations and inform the user before they are recognized. 

6. Recognition rather than recall 

Interactive systems should be designed to minimize the user’s workload by develop- 
ing options, elements, and actions that are visible to the user. Interactive systems 
must provide instructions and ideas that facilitate operation and efficient use. This 
concept implies that the system interface should allow the user to recognize infor- 
mation, rather than store it with them. 

7. Flexibility and efficiency of use 

This principle states that interaction with the system should be the same for novice 
and experienced users, and that experienced users should use keyboard shortcuts to 
accomplish tasks. Nevertheless, all users should be the same, as novices perform 
tasks in a standard way. 

8. Aesthetics and minimalistic design 

This principle states that an interactive system should develop useful information 
and not redundant irrelevant information. Unnecessary information can override 
important information and reduce the visibility of the content. Developers should 
identify important information that should be shown to users. 

9. Help users recognize, diagnose, and recover from errors 

This principle states that error messages should be expressed in human-understandable 
language and offer solutions to user errors. The aesthetics of error messages also play 
an important role in the design of the interface. Responses should be presented in 
an understandable form so that the user can correct the error. 

10. Help and documentation 

The system interface should be user-friendly and should not require additional ex- 
planation. However, appropriate documentation may be required to enable users to 
complete tasks. Help and documentation elements should be easy to find and help 
the user accomplish the task. Specific steps should be specified to assist the user in 
completing the task. 


1.3.3. Coursera and Udemy 


Coursera 


Coursera was founded in 2012 by Daphne Koller and Andrew Ng to give a good 
learning experience to the students around the world. Today, Coursera is a global 
online learning platform that delivers online courses and degrees from leading uni- 
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versities and companies to anyone, anywhere. Coursera was certified by Corporation 
B in February 2021, which means not only legal accountability for shareholders, but 
also reduce barriers to access to a world-class education for all. Coursera partners 
with more than 200 leading universities and companies to bring flexible, accessible, 
work-related online learning to individuals and organizations around the world. User 
interface of coursera is attached in Appendix A.1 


Udemy 


Udemy founder Ellen Barry had little access to education until she got a computer. 
Changing her life through online learning, Ellen partnered with co-founders of Udemy 
to achieve a common goal of providing quality education to all. Udemy, guided by 
its mission, enables anyone, anywhere in the world, to create courses and sell their 
knowledge. This access opens up new opportunities for people and organizations 
around the world to gain the knowledge and skills they need to prepare for the 
future. User interface of Udemy is attached in Appendix A.2 


1.3.4 Similar MOOC platforms 


EdX 


EdX was founded in 2012 by Harvard University and the Massachusetts Institute of 
Technology as a nonprofit organization. However, in 2021, edX was acquired by 2U 
for $800 million and ceased to be a nonprofit organization. edX is the second largest 
MOOC platform in the world with more than 42 million students and 7 million new 
students. edX offers around 3,550 courses and cooperates with more than 100 uni- 
versities, and offers many types of certification programs. 


Udacity 


Udacity has abandoned its former identity as a MOOC platform. It has part- 
nered with technology companies to create Nanodegrees, which prepare students for 
technology-based professions. A nanodegree costs about $1,000 and can take several 
months to complete. Udacity has stopped publishing free courses, but its catalog 
still contains a number of courses that can be accessed for free. In 2021, Udacity im- 
proved its offerings to businesses, tripling the number of scholarships while reducing 
the number of free courses. 


Khan Academy 


Khan Academy’s organizational mission is based on the belief that education is a hu- 
man right. To that end, it offers completely free online courses that include videos, 
exercises, and individual learning panels. Unlike many other MOOC providers, Khan 
Academy primarily focus on 12th grade students and offers courses in pre-algebra, 
English, chemistry, American government, and civics. In addition, Khan Academy 
provides resources for teachers and parents to implement standards-aligned materi- 
als and monitor student progress. Web-based tools allow users to communicate with 
students and teachers around the world. 


Tools used in this study: 


The tools used in the survey are Coursera, Udemy and Google forms. 
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Coursera and Udemy are MOOC platforms where the users are enrolled in a specified 
course and get acquainted with the platforms by completing their tasks. 

Google forms are used for the survey questionnaire, recording the responses and 
generating graphs for the responses recorded for the questionnaire. 


1.4 Scope of the thesis 


The scope of this study covers selected MOOC platforms Coursera and Udemy. This 
study is limited to assessing the usability components: Effectiveness, Efficiency, Sat- 
isfaction level, and user experience of two MOOC platforms(Coursera and Udemy). 
The survey results are for free courses only and do not apply to fee-based courses, 
so the article results cannot be generalized. The survey evaluation is taken to get 
a better user experience with the platforms. A survey was conducted among a few 
respondents by relating the usability heuristics with the selected MOOC platforms, 
and the conclusion is based on the survey results. 


1.5 Outline 


This thesis mainly focuses on user satisfaction, effectiveness, efficiency, and usability 
heuristics compared with the selected MOOC platforms Coursera and Udemy. The 
first research question is regarding the user satisfaction level, efficiency, and effec- 
tiveness of the MOOC platforms. These are calculated by using the respondent’s 
time of completion for the specific tasks given to them and the feedback of each 
task performed by the respondents. The second research question is based on a 
comparative analysis of user experience with the MOOC platforms through a survey 
questionnaire for the fifteen respondents. The survey results are concluded from the 
graphical representation. 


Chapter 2 


Related Work 


This section gives an overview of the research papers which are relevant to our re- 
search study. 


Nielsen [16] has listed ten general principles for user interface design, called usability 
heuristics. These include visualization of the system state, alignment of the system 
with the real world, user control and freedom, consistency and standards, error pre- 
vention, recognition rather than memorization, flexibility and usability, aesthetics 
and minimal design, helping users identify, diagnose, and correct errors, and help 
and documentation. This paper inspired to use of heuristic usability principles ap- 
plied to the selected MOOC platform to measure the user experience. 


Conache et al. [5] discuss the usefulness of MOOC platforms to revitalize teaching 
and learning and create new virtual spaces for educational exchange. The main 
points describing MOOC platforms are gathered. The factors analyzed have been 
divided into three categories: business model, course design, and popularity. A de- 
tailed comparison of the four platforms is also given based on the data collected from 
each platform. This research paper inspires the data collection idea. 


Zhou |27| explains that massive open online courses (MOOCs) represent a new in- 
tellectual and technological approach and personalized teaching and learning philos- 
ophy. The prominence of MOOCs in the development of distance education signifies 
the arrival of a new era, that of open learning. The usability of MOOCs platforms 
was evaluated quantitatively. The study focused on the actual user experience rather 
than traditional technical issues and features. Methodologically, several improve- 
ments over previous studies were suggested. This study inspired to focus on user 
experience in MOOC platforms rather than the traditional educational systems. 


Bevana et al. [4] argue that usability resides in the interaction between the user 
and the product or system and can only be accurately measured by assessing user 
performance, satisfaction, and acceptance. Usability can change as the product’s 
characteristics, system, user, task, or environment change. A product is not us- 
able or unusable by itself but has characteristics that determine its suitability for 
a particular user, task, or environment. This paper helped get an idea about the 
interactiveness between the system and the user interface. 


Espada et al. [6] state that a common feature of these courses is the use of assess- 
ment tests to demonstrate course success and student knowledge acquisition. This 
analysis considers most of the characteristics and elements that influence the on- 
line user experience and highlights the characteristics that can significantly impact 
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MOOC platforms. This study inspired us to consider calculating respondents’ skills 
or knowledge. 


Pireva et al. [21] focus on a multi-point comparison of advanced LMSs. They also 
explore the behavior and experience of users of different LMSs and MOOCs for the 
first time by observing users’ facial expressions. From this paper, a comparison of 
state of the art in multiple points and how to examine the behavior of the respon- 
dents while performing the tasks can also make a difference is known. 


Shubina [23] explains how she determined and compared user satisfaction of major 
MOOC platforms. Usability testing methodology was applied to study user satisfac- 
tion of three major MOOC providers. During the study of this paper, the idea of the 
history of distance learning, the MOOC phenomenon, usability theory, and usability 
testing methods helped to understand the chosen topic better. 


Oktavia et al. [19] discussed the definition of critical features of MOOCs based on the 
educational perspective of online learning activities and RASE (Resource Activity 
Support Evaluation) and how this platform can be used as a global opening. From 
the learner’s perspective, these features included in edX and Coursera have similar 
functions and activities. The components of EBP pedagogy (resources, activities, 
support, and assessment) were compared with the operational activities performed. 
We have been inspired by the features and activities done using the MOOCs from 
the paper. 


Gattupalli and Reddivari |7| describe methods for empathizing with usability heuris- 
tics. The usability heuristic compares free versions of canvas and google-classroom 
to measure user satisfaction. The study focuses on user satisfaction when using LMS 
platforms such as Canvas and google classroom and uses usability heuristics to mea- 
sure user satisfaction. This paper inspired how to use the techniques to compare the 
usability heuristics and the measures. 


Sukamolson [24] discusses the main basic concepts of quantitative research. He ex- 
plains types of research, the definition of quantitative research, types of quantitative 
research and prerequisites for using or not using quantitative methods, benefits of 
quantitative research, standard methods and designs, and common misconceptions. 
Criteria for evaluating quantitative research programs are also provided. The main 
topics covered are the essential prerequisites for quantitative research and the mis- 
conceptions that many researchers encounter. This paper is the inspiration for taking 
the quantitative research method. 


Johansson and Frolov [10] conducted a study to develop a list of recommendations 
and a usability checklist to help evaluate the usability of user interfaces for Massive 
Open Online Courses (MOOCs). Interviews had conducted to understand their use 
and identify design priorities for evaluating MOOC interfaces. Then, using Jacob 
Nielsen’s heuristic method, the author’s ten heuristics were used to examine the us- 
ability issues of these design focuses. The study came to two main conclusions. First, 
MOOGs are used differently from traditional university courses, with a different user 
experience and approach. Second, usability checklists need to be adaptable and 
updateable to respond to changes in the MOOC usage environment. Using Jacob 
Nielsen’s heuristics to develop and checklist is taken from this paper. 
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Korableva et al. [11] studied the advantage of online courses over traditional educa- 
tion. Their research on why students fail in online courses has yielded mixed results, 
but one of the factors identified is interface design and the issue of interface impact 
on the user. This paper studies two different online MOOC platforms to identify 
factors that increase usability and develop a methodology for determining the opti- 
mal design solution. A comparative analysis of two MOOC platforms (Coursera and 
Open Education) was used to identify optimal behaviours and to understand which 
details of online platform design to pay attention to and which not to. This paper 
inspired a comparative study between the two MOOC platforms. 


Shubina |23] compared user satisfaction levels of representative MOOC platforms. A 
quantitative research method was used in this paper. Usability testing methods were 
applied to investigate user satisfaction of three large-scale online course providers. 
Usability was assessed for each MOOC platform to answer the main research ques- 
tion. To carry out this work, the authors studied the history of the development 
of distance education, the phenomenon of MOOCs, usability theory, and usability 
testing methods and techniques to broaden their knowledge on the chosen topic. In 
this paper, the quantitative method is considered, and survey data of users’ age, 
gender and level of education helped analyse the user’s level of knowledge and skills. 


Chapter 3 


Method 


This chapter represents the methodology used to conduct the experiments in this the- 
sis. The outcome of this chapter is a comparative evaluation of the MOOC platforms 
- Coursera and Udemy - comparing them using usability heuristics and examining 
the effectiveness, efficiency, and satisfaction of the MOOC platforms. A comparative 
evaluation is a process of collecting data and analyzing the characteristics of different 
products or applications to make a decision and achieve a certain goal. Users were 
given specific tasks on the platforms and asked to answer a questionnaire based on 
their platform experience. The questionnaire was based on usability heuristics. The 
usability heuristic is described in more detail in Section 1.3. 


3.1 Quantitative Research Method 


Quantitative research method |24] involves the numerical representation and handling 
observations, and the phenomena reflected in the observations can be interpreted. 
Quantitative research is essentially the collection of numerical data to explain a phe- 
nomenon, and it seems immediately appropriate to answer a particular question using 
quantitative methods. 


Quantitative research is of different types: survey research, experimental research, 
correlational research, and causal-comparative research. In this paper, survey re- 
search is used by forming a questionnaire design to measure which MOOC platform 
(Coursera and Udemy) gives a better experience for the user using the heuristic prin- 
ciples. 


In this paper, similar tasks are allocated to fifteen users, and the timings of the tasks’ 
performed by the users are noted. These observations from the users are the numer- 
ical data, which is used to calculate the effectiveness and efficiency of the MOOC 
platforms (Coursera and Udemy). 


Survey research method 


The survey research method is defined as the collection of information from a sam- 
ple through the responses of individuals to questions. This type of research can use 
various methods, including participant recruitment, data collection, and a variety of 
tools. The survey research method can involve quantitative research strategies, such 
as questionnaires, and qualitative research strategies, such as open-ended questions 
or mixed methods at the same time.[22] 
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The main objective of this type of survey research method is to obtain information 
describing the characteristics of a sample of target individuals over a relatively short 
period. These surveys are usually conducted by mail or through a google form to 
obtain opinions that form the basis of a product for a group. 


3.2. Scientific Methods 


1. The scientific method used for RQ1 : 


RQ1 will be scientifically answered by calculating the two MOOC platforms’ 
effectiveness, efficiency, and satisfaction level from the survey data collected. 


Effectiveness : 


Effectiveness is measured using the completion rates and the number of errors. 
Completion is the percentage of the tasks which the users complete successfully. 
Effectiveness will be calculated by measuring the completion rate of the tasks 
given to the user. 


Efficiency : 


Efficiency helps calculate the user’s time spent completing the tasks. Both 
efficiency and Effectiveness are directly related, as efficiency is the user effec- 
tiveness divided by the time spent by the user, and it is time-based efficiency. 
Efficiency will be calculated by measuring the time taken by the user to com- 
plete the specific tasks. 


Satisfaction Level : 


Task level satisfaction helps measure the users’ impression when confronted 
with the system while working on it. Satisfaction level will be obtained by 
collecting the user feedback through a Likert scale rating[13]. 


A Likert scale is used mainly in questionnaires to determine participants’ level 
of preference or agreement with a statement or set of words. Likert scales are 
non-comparative assessment methods and are unidimensional (measuring only 
one characteristic). Respondents are asked to represent their level of agreement 
with the statement on a continuous scale. |2| 


Very Difficult | Difficult | Somehow Difficult | Neutral | Somehow Easy | Easy | Very Easy 


Likert Values 


(1) 


(2) 


(3) 


(4) 


(5) 


(7) 


Tasks 


Oo 


oO 


Oo 


Oo 


Oo 


Oo 


Table 3.1: Likert Scale Rating for each task 


2. The scientific method is used for RQ2 : 


RQ2 will be answered by using statistical measures such as variance and stan- 
dard deviation. 


3.3. Performing tasks in MOOC platforms: 15 


3.3 Performing tasks in MOOC platforms: 


This section explains the details of the tasks given to the respondents and how the 
experiments were conducted. Tasks descriptions were explained to the respondents, 
and eight tasks were given. 

The respondents were given eight tasks to perform on both MOOC platforms and 
the timings of each completed task while performing the tasks are noted. In both 
MOOC platforms,similar tasks were given to the respondents. Each task given to 
the respondents on both platforms is explained in detail. 

Task 1 

The user’s task is to open the browser and log in/signup for the MOOC platforms 
(Coursera, Udemy). 

Task 2 

The user’s task is to open the search bar in the MOOC platforms and search for the 
specified course (Machine learning). 

Task 3 

The specified course in the MOOC platform is opened, and there the users need to 
check out the course content /info. 

Task 4 

The user’s task is to enroll in the course and play the introduction video of the course. 
Task 5 

The user’s task is to complete the quiz based on the introduction video and submit 
the quiz. 

Task 6 

The user’s task is to update/upload the profile. 

Task 7 

The user’s task is to check for the notifications on the MOOC platforms. 

Task 8 

The user’s task is to share the course using social network which the MOOC plat- 
forms provide. 


After tasks were completed, the respondent’s completion time for the tasks was 
noted to calculate the effectiveness and efficiency. Furthermore, the respondents 
were given the Likert scale to measure the satisfaction level of each platform. The 
respondents will give the Likert scale values for each task performed on Coursera 
and Udemy. The results obtained from the respondents will be used to measure 
satisfaction levels. 

The effectiveness, efficiency, and satisfaction level are calculated and mentioned in 
chapter 5. 


3.4 Survey Evaluation 


A survey is a research method defined by collecting data from a sample of respon- 
dents to obtain and record users’ opinions. Our survey questionnaire consists of 
closed questions, Likert scale questions, and multiple-choice questions. The survey 
was conducted using Google Forms. 
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The questionnaire covered the tasks presented in section 4.1. The task questions 
corresponded to the usability heuristics presented in section 4.2. The data collected 
to calculate satisfaction level, effectiveness, and efficiency are tabulated in Chapter 5. 
The survey results were converted into statistical data. Most of the statistics con- 
sisted of pie charts and bar graphs. The respondent’s details are kept confidential. 


3.5 Profile of Respondents 


This section provides details about the respondents who participated in the survey. 
The participant’s details are anonymous. Fifteen participants were included in the 
survey; four were familiar with the course, and the remaining were familiar with 
both platforms. The respondents’ gender, age, and education were asked. This 
information was used to determine the skill and knowledge level of the users. Of 
the respondents,60% were male and 40% were female. Participant’s ages were in the 
range 18-25 years. The respondents have a Bachelor’s or a Master’s degree. The 
level of education of one respondent is a Master’s degree, and the remaining were 
Bachelor’s degree respondents. 


Number of responses (N=15) | Percentage (100%) 
Gender 
Male 9 60 
Female 6 AO 
Age 
18-21 9 60 
22-25 6 AO 
Level of Education 
Bachelor’s degree 14 93 
Master’s degree 1 7 


Table 3.2: Participants profile 


Chapter 4 


Data Collection 


Data collection is the systematic process of collecting observations or measurements. 
A survey method is used in the thesis to collect data, and all questions asked to the 
respondents in the survey are mentioned in this section. 


4.1 Survey Questionnaire 


This section contains screenshots of the questions in our survey. The survey questions 
are a combination of Likert scale questions and closed-ended questions. 


1. Are you familiar with any of these MOOC platforms mentioned below? * 
Coursera 
Udemy 
Both 


Other... 


Figure 4.1: Survey Question 1 
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2. Which of the platform is easy to login/sign up? * 


Coursera 
Udemy 
Both 


Other... 


Figure 4.2: Survey Question 2 


3. Which platform do you find easy to search the courses? * 


Coursera 
Udemy 
Both 


Other... 


Figure 4.3: Survey Question 3 


4. In Which platform do you face less difficulty in enrolling the courses? * 


Coursera 
Udemy 
Both 


Other... 


Figure 4.4: Survey Question 4 
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5. Did you face any user control issues in performing task 4 like interruptions in playing a video * 


in any of these platforms? 


Coursera 
Udemy 
Both 


Other... 


Figure 4.5: Survey Question 5 


6. Which platform is easier to navigate through the questions of the quiz to complete the quiz? * 


Coursera 
Udemy 
Both 


Other... 


Figure 4.6: Survey Question 6 


20 Chapter 4. Data Collection 
7. While uploading or updating the profile Pe the MOOC platforms which one is faster 
and efficient? 
Coursera 
Udemy 
Both 


Other... 
Figure 4.7: Survey Question 7 


8. In which MOOC platform it is easier to find the notification icon or label similar to other 
social media (like Facebook, LinkedIn)? 


Coursera 
Udemy 
Both 


Other... 


Figure 4.8: Survey Question 8 


9. Which platform gave a better share experience while sharing the course to others? * 
Coursera 
Udemy 
Both 


Other... 


Figure 4.9: Survey Question 9 
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10. Which MOOC platform do you think has a better help and documentation section? * 


Coursera 
Udemy 
Both 


Other... 


Figure 4.10: Survey Question 10 


11. Which platform showed a clear guidance while you are working on like displaying error 
messages while a wrong step is taken? 


Coursera 
Udemy 
Both 


Other... 


Figure 4.11: Survey Question 11 


12. Enter your age? * 


Below 18 
18-21 
22420 


Above 25 


Figure 4.12: Survey Question 12 


22 Chapter 4. Data Collection 


13. Enter your gender? * 


Male 
Female 


Other... 


Figure 4.13: Survey Question 13 


14. What is your education level? * 


Bachelor’s degree 
Master's degree 


Other... 


Figure 4.14: Survey Question 14 


15. By considering overall experience while performing the tasks which platform do you think * 


has the better user experience? (Coursera) 


(Strongly Disagree), Disagree, Somewhat Disagree, Neither Agree or Disagree, Somewhat Agree, Agree, Strongly Agree, (Don't 
Know) 


1 24 3 4 5 6 7 8 


Strongly Disagree S O O O O O O O Don't Know 


Figure 4.15: Survey Question 15 
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16. By considering overall experience while performing the tasks which platform do you think * 


has the better user experience? (Udemy) 


(Strongly Disagree), Disagree, Somewhat Disagree, Neither Agree or Disagree, Somewhat Agree, Agree, Strongly Agree, (Don't 
Know) 


1 2 3 4 5 6 7 8 


Strongly Disagree O O O O O O O O Don't Know 


Figure 4.16: Survey Question 16 
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Survey Questionnaire 


Usability Heuristics 


1. Are you familiar with any of these 
MOOC platforms mentioned below? 


To know the user’s familiarity with 


MOOC platforms 


2. Which platform is easy to login/sign 
up for? 


a. Visibility of system status 
b. Aesthetic and minimalist design 


3. Which platform do you find easy to 
search for the courses? 


c. Consistency and standards 
d. Recognition rather than recall 


4. In which platform do you face less 
difficulty in enrolling the courses? 


e. Match between the system and the 
real world 


5. Did you face any user control issues 
in performing task 4, like interruptions 
in playing a video on any of these plat- 
forms? 


f. User control and freedom 


6. Which platform is easier to navigate 
through the quiz questions to complete 
the quiz? 


a. Visibility of system status 
g. Error prevention 


7. While uploading or updating the 
profile picture of the MOOC platforms, 
which one is faster and more efficient? 


a. Visibility of system status 


8. In which MOOC platform is it eas- 
ier to find the notification icon or label 
similar to other social media (like Face- 


book, LinkedIn)? 


e. Match between the system and the 
real world 


9. Which platform gave a better share 
experience while sharing the course 
with others? 


h. Flexibility and efficiency of use 


10. Which MOOC platform do you 
think has a better help and documen- 
tation section? 


i. Help and document 


11. Which platform showed clear guid- 
ance while you are working on, like dis- 
playing error messages while a wrong 
step is taken? 


j. Help users recognize, diagnose and 
recover from errors 


12. Enter your age? 
13. Enter your gender? 
14. What is your education level? 


To understand the degree of user skills 
and knowledge 


15. By considering the overall experi- 
ence while performing the tasks, which 
platform do you think has the better 
user experience? (Coursera) 


Overall user experience with the plat- 
form 


16. By considering overall experience 
while performing the tasks which plat- 
form do you think has the better user 
experience? (Udemy) 


Overall user experience with the plat- 
form 


Table 4.1: Mapping usability heuristics to survey questions 


Chapter 5 


Results and Analysis 


This chapter contains the results of the survey conducted on fifteen users. Four 
of them were familiar with Coursera, and the remaining were familiar with both 
platforms. The threshold is determined by the average number of times performance 
is measured. To calculate effectiveness and efficiency, the time it takes each user to 
complete a task is measured. The threshold value is the average value of each task 
performed by every user. The values which are higher than the threshold value are 
not included in the usability index calculation. For measuring usability, effectiveness 
and efficiency are measured, and satisfaction is measured using task-level (SEQ). 


5.1 Task timings for both MOOC platforms 


Coursera 
The below table 5.1 shows each task completed by individual users of the Coursera 
platform. 


Users Task1 | Task2 | Task3 | Task4 | Task5 | Task6 | Task7 | Task8 
1 15 38 108 16 257 147 33 76 
2 18 19 187 10 264 107 34 12 
3 14 39 242 15 317 171 31 121 
4 60 20 187 12 344 84 32 10 
5 15 38 108 18 257 147 33 76 
6 81 21 247 13 256 81 31 20 
7 62 20 243 16 345 74 31 11 
8 80 20 188 20 286 72 21 10 
9 62 20 248 10 345 74 31 11 
10 85 21 242 13 236 71 12 10 
11 04 20 135 14 341 81 30 10 
12 116 14 243 15 301 65 31 10 
13 92 21 244 16 276 65 34 10 
14 102 12 222 10 291 62 21 10 
15 112 10 221 12 302 71 12 12 
Threshold Value | 64.53 22.2 204.33 | 14 294.53 | 91.46 27.8 27.26 
Table 5.1: Task Timings (in sec) of Coursera 
Effectiveness : 


Effectiveness will be calculated by measuring the completion rate of the tasks given 
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to the user. 


Number of tasks completed successfully 


Effectiveness = x 100 


Total Number of tasks undertaken 
= 69/120 « 100 


Effectiveness = 57.5% 
(Ssh) 


Number of tasks completed successfully mean the tasks completed within the thresh- 
old time. 


Efficiency : 


Efficiency will be calculated by measuring the time taken by the user to complete 
the specific tasks. 


Dre 


Efficiency = == 5.2 
ciency = +p ( 
Efficiency of Coursera = 0.027goals/sec. (5.3) 


Udemy 
The below table 5.2 shows each task completed by individual users of the Udemy 
platform. 


Users Task1 | Task2 Task3 Task4 | Task5 | Task6 | Task7 Task8 
1 67 41 141 19 166 159 12 88 

2 63 40 180 15 181 62 57 28 

3 10 41 145 22 166 209 10 80 

4 64 30 242 19 340 61 45 25 

5 57 Al 141 19 166 159 12 88 

6 73 34 226 18 351 121 24 23 

7 56 29 302 20 289 61 25 24 

8 59 59 226 22 351 59 25 20 

9 56 29 302 21 289 61 25 24 

10 54 30 184 18 161 94 15 15 

11 71 23 141 19 242 144 20 9 

12 102 21 186 19 292 64 24 13 

13 102 17 272 10 222 116 25 12 

14 100 15 194 15 221 65 15 12 

15 65 13 201 14 271 59 16 15 
Threshold value | 66.6 30.86667 | 205.5333 | 18 247.2 | 99.6 23.33333 | 31.73333 


Table 5.2: Task Timings (in sec) of Udemy 


Effectiveness: Effectiveness will be calculated by measuring the completion rate 


5.1. Task timings for both MOOC platforms Af 


of the tasks given to the user. 


Number of tasks completed successfully 


Effecti = 1 
ene Total Number of tasks undertaken a 
= 70/120 « 100 
Effectiveness = 58.3% 
(5.4) 
Efficiency : 


Efficiency will be calculated by measuring the time taken by the user to complete 
the specific tasks. 


ym 


Efficiency = ————_— 5.5 
ciency = 7p (5.5) 
Efficiency of Udemy = 0.020goals/sec. (5.6) 


From the above calculations, the total successful tasks of Coursera is 69 and for 
Udemy is 70. The difference between Coursera and Udemy’s effectiveness and effi- 
ciency is negligible from the analysis. 


Satisfaction through SEQ(Single Ease Questionnaire): 


The SEQ (Single Ease Questionnaire) is another type of satisfaction questionnaire, 
but this one is specific to the user’s task and not to the system as a whole SEQ has 
a single question, "How easy is this task?" The SEQ is given to the participant, who 
expresses his or her opinion about the task. This question is based on a seven-point 
Likert scale (1-7). Although many other questionnaires are designed at the task or 
system level, this questionnaire is easy to use and has proven to be very effective. 


Task-level satisfaction is measured by calculating the values of these scales. The fol- 
lowing scale represents the scores (1-7) for Very Difficult, Difficult, Fairly Difficult, 
Neutral, Fairly Easy, Easy and Very Easy. 


Liteon’ Valiace Very Difficult | Difficult | Somehow Difficult | Neutral | Somehow Easy | Easy | Very Easy 
(1) (2) (3) (4) (5) (6) | (7) 
Taskl fe) fo) oO fo) fo) fe) fo) 
Task2 fe) fo) fo) fo) fo) fe) fo) 
Task3 fo) fo) fe) fo) fe) fe) fe) 
Task4 fe) fo) fo) Oo oO fo) fo) 
Task5 fo) fe) fe) fo) fo) fe) fe) 
Task6 fo) fo) fe) fo) fe) fo) fo) 
Task7 fo) fe) fe) fo) fo) fe) fe) 
Task8 fo) fo) fe) fo) oO fo) fo) 


Table 5.3: Likert Scale Rating 


The above table 5.3 is given to the user for rating each task after all the tasks are 
completed. According to Coursera and Udemy’s Likert scale, participants’ responses 
to each task are listed below. 
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5.1.1 Coursera Platform Likert Scale Rating 


Task1 | Task2 | Task3 | Task4 | Task5 | Task6 | Task7 | Task8 


= 
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ie) 
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Table 5.4: Coursera Likert Scale Rating 


Calculation Method 

Users are multiplied by the number of cells in the scale and divided by the total 
number of users. This method is applied to each task. The average ratio is then 
calculated for each task. Finally, all responses for each task are added together and 
divided by the total number of users. 

Calculations for Coursera Platform 


Task 1: 
8*x7+7*6 
= 98/15 
= 6.53( Easy) 
(a0) 


Task 2: 
5*7+10*6 
= 05/15 
= 6.33( Easy) 
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Task 3: 
4*64+9*5+4+2«4 
= 77/15 
= 5.13(FairlyEasy) 


Task 4: 
8*5+6%*44 %3 
= 67/15 
= 4.46( Neutral) 
(5.10) 


Task 5: 
5*5+9*44+1«3 
= 64/15 
= 4.26( Neutral) 
(ook) 


Task 6: 
l*7+7*64+7%*5 
= 84/15 
= 5.6(FairlyEasy) 
(5.12) 


Task 7: 
4*7+11*6 
= 94/15 
= 6.26( Easy) 
(5.13) 


Task 8: 
4*7+11*6 
= 98/15 
= 6.53(Easy) 
(5.14) 


Total: 


6.3 + 6.33 + 5.13 + 4.46 + 4.26 + 5.6 + 6.26 + 6.533 
= 5.63(FairlyEasy) 
(5.15) 


30 Chapter 5. Results and Analysis 


5.1.2 Udemy Platform Likert Scale Rating 


Users | Task] | Task2 | Task3 | Task4 | Task5 | Task6 | Task7 | Task8 
1 6 6 5 5 o 6 7 6 
2 6 6 Es) 4 4 6 6 6 
3 7 6 4 3 2 Db 6 7 
4 6 6 6 4 4 3 y 6 
) 6 6 D 4 ) 6 6 6 
6 6 7 ) 4 4 6 6 i 
ri 7 ¢ 4 5D 4 6 6 6 
8 6 6 OD D 4 6 6 7 
9 7 6 Es) OD 4 Es) 6 6 
10 7 6 6 4 OD o 6 6 
11 ¢ 6 6 4 4 D 6 7 
12 7 6 D 5 4 Es) 6 6 
13 6 7 . 4 Es) 6 7 6 
14 6 " 3) ES) 4 a) 6 ¢ 
15 7 6 Es) 4 5 6 6 6 
Table 5.5: Udemy Likert Scale Rating 
Calculations for Udemy Platform 

Task 1: 

8*6+7%*7 

= 97/15 
= 6.46( Easy) 
(5.16) 

Task 2: 

11*64+4x«7 

= 94/15 
= 6.26(Easy) 
(5.17) 

Task 3: 


2*4+10*5+3%6 
= 96/15 
= 5.06(FairlyEasy) 
(5.18) 
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Task 4: 
1*34+8%*4+6«5 
= 92/15 
= 6.13(Fasy) 
(5.19) 


Task 5: 
2%*34+9*44+4x5 
=62/ 15 
= 4.13( Neutral) 
(5.20) 


Task 6: 
8*6+7%*5 
= 83/15 
= 5.53(FairlyEasy) 
(5.21) 


Task 7: 
12*64+3*7 
= 93/15 
= 6.20( Easy) 
(5.22) 


Task 8: 
10*6+5*7 
= 75/15 
= 5(FairlyEasy) 
(5.23) 


Total 


Total = 6.46 + 6.26 + 5.06 + 6.13 4+ 4.13 +5.534+6.2+5 
= 5.61(FairlyEasy) 
(5.24) 


The above calculations of Coursera and Udemy are Fairly Easy for overall tasks, 
where Coursera and Udemy got 6.63 and 6.61 values, respectively. 
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The answers to the questionnaire are displayed in the form of graphs. 


1. ARE YOU FAMILIAR WITH ANY OF THESE MOOC 
PLATFORMS MENTIONED BELOW? 


Coursera 
Udemy mw Udemy 
0% mg Both 


Figure 5.1: Responses for survey question-1 


From the figure 5.1, most of the respondents have used both Coursera and Udemy. 
27% were familiar with Coursera, and 73% were familiar with both platforms. 


2. WHICH OF THE PLATFORM IS EASY TO LOGIN/SIGN 
UP? 


Coursera 


Both mw Udemy 


53% @ Both 


13% 


Figure 5.2: Responses for survey question-2 


From the figure 5.2, 34% of the respondents found Coursera easy to log in, and 
13% of the respondents found Udemy easy to log in. At the same time, 53% of the 
respondents found both platforms for easy login and signup. 
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3. WHICH PLATFORM DO YOU FIND EASY TO SEARCH THE 
COURSES? 


= Coursera 
m Udemy 
@ Both 


27% 


Figure 5.3: Responses for survey question-3 


From the figure 5.3, 46% of respondents found Coursera easy to search the courses 
on the platform. In contrast, only 27% respondents found Udemy easy to search the 
courses, and 27% respondents found both platforms were easy to search the courses. 


4. IN WHICH PLATFORM DO YOU FACE LESS 
DIFFICULTY IN ENROLLING THE COURSES? 


Udemy Both 
7% 


None 
40% Coursera 


mw Udemy 
= Both 


g None 


Figure 5.4: Responses for survey question-4 


From the figure 5.4, 53% of the respondents found Coursera is better in enrolling 
the courses and 7%of the respondents found Udemy is better, whereas 40% respon- 
dents found both platforms have no difficulty enrolling the courses. 
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5. DID YOU FACE ANY USER CONTROL ISSUES IN 
PERFORMING TASK 4 LIKE INTERRUPTIONS IN PLAYING 
A VIDEO IN ANY OF THESE PLATFORMS? 


Both 
13% 


Udemy 
13% 


Figure 5.5: Responses for survey question-5 


From the figure 5.5, 40% of respondents did not find any issues while playing the 
course videos. 34% of respondents found user control issues in Coursera and 13% in 
Udemy, whereas 13% in both the platforms. 


6. WHICH PLATFORM IS EASIER TO NAVIGATE THROUGH 
THE QUESTIONS OF THE QUIZ TO COMPLETE THE QUIZ? 


Both 
13% 


= Coursera 


Udemy m= Udemy 


27% m@ Both 


Figure 5.6: Responses for survey question-6 


From the figure 5.6, 60% of the respondents found Coursera is easier to navigate 
through the quiz and 27% of them found Udemy is easy, and 13% of the respondents 
found both platforms were better in navigating the quiz. 
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7. WHILE UPLOADING OR UPDATING THE PROFILE 
PICTURE OF THE MOOC PLATFORMS WHICH ONE IS 
FASTER AND EFFICIENT? 


Both 
20% 


= Coursera 
m Udemy 
@ Both 


Figure 5.7: Responses for survey question-7 


From the figure 5.7, 47% of the respondents found that Coursera is faster and 
more efficient in updating or uploading the profile. Furthermore, for Udemy, it is 33 
%, whereas 20% of the respondents found both are faster in updating the profile. 


8. IN WHICH MOOC PLATFORM IT IS EASIER TO FIND THE 
NOTIFICATION ICON OR LABEL SIMILAR TO OTHER 
SOCIAL MEDIA (LIKE FACEBOOK, LINKEDIN)? 


= Coursera 
m Udemy 
@ Both 


'Yy 
33% 


Figure 5.8: Responses for survey question-8 


From the figure 5.8, 34% respondents agreed that Coursera notifications are sim- 
ilar to other social media, and 33% of respondents agreed with the Udemy. The 
remaining 33% found both the platforms were similar to other social media plat- 
forms. 
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9. WHICH PLATFORM GAVE A BETTER SHARE EXPERIENCE 
WHILE SHARING THE COURSE TO OTHERS? 


Both = Coursera 


46% m Udemy 


@ Both 


27% 


Figure 5.9: Responses for survey question-9 


46% of the participants found both platforms have better share experience from 
the figure 5.9. And Coursera and Udemy both platforms got 27% respondent per- 
centage for the good share experience. 


10. WHICH MOOC PLATFORM DO YOU THINK HAS A 
BETTER HELP AND DOCUMENTATION SECTION? 


Both 
27% 


= Coursera 
m Udemy 
@ Both 


Udemy 
33% 


Figure 5.10: Responses for survey question-10 


From the figure 5.10, 40% of respondents were able to think Coursera has a better 
help and documentation section, whereas 33% of people found Udemy. Furthermore, 
27% thought both platforms had good help and documentation sections. 


5.1. Task timings for both MOOC platforms 


11. WHICH PLATFORM SHOWED A CLEAR GUIDANCE 
WHILE YOU ARE WORKING ON LIKE DISPLAYING 
ERROR MESSAGES WHILE A WRONG STEP IS TAKEN? 


Udemy 
27% 


Figure 5.11: Responses for survey question-11 
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33% of the participants found Coursera showed better error-prevention methods 
from the figure 5.11. Moreover, 27% of the respondents found it in Udemy, and 
27% of the participants found both have error prevention methods. Whereas 7% 
respondents found both platforms do not have any error preventing messages while 


a wrong step is taken. 


12. ENTER YOUR AGE 


Figure 5.12: Responses for survey question-12 


From the figure 5.12, 60% of the people who participated in this survey are aged 


18-21 and 40% of them are 22-25. 
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13. ENTER YOUR GENDER 


Female 


= Male 


= Female 
Male 
60% 


Figure 5.13: Responses for survey question-13 


From the figure 5.13, 40% of the respondents were female, and 60% of them were 
male. 


14. WHAT IS YOUR EDUCATION LEVEL? 


Bachelor's 
Degree 
93% @ Bachelor's Degree 
Master's mw Master's Degree 
Degree 


7% 


Figure 5.14: Responses for survey question-14 


From the figure 5.14, 93% of the respondents who participated in the survey are 
Bachelor degree level, and 7% are Mater degree level respondents. 
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15. By considering overall experience while 
performing the tasks which platform do you think 
has the better user experience? (Coursera) 
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Figure 5.15: Responses for survey question-15 


From figure 5.15, Users were asked to rate their level of satisfaction when using 
Coursers on a scale of 1 to 8. Where (Strongly Disagree, Disagree, Somewhat Dis- 
agree, Neither Agree or Disagree, Somewhat Agree, Agree, Strongly Agree, Don’t 
Know: 1-8 respectively.) 


Likert Scale | 1/2/)3/4/5]6/17)]8 
Values 0'0;0/1/0/7}7)]0 


Table 5.6: Likert scale of survey question-15 


16. By considering overall experience while 
performing the tasks which platform do you think 
has the better user experience? (Udemy) 
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Figure 5.16: Responses for survey question-16 


From figure 5.16, Users were asked to rate their level of satisfaction when using 
Udemy on a scale of 1 to 8. Where (Strongly Disagree, Disagree, Somewhat Disagree, 
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Neither Agree or Disagree, Somewhat Agree, Agree, Strongly Agree, Don’t Know: 
1-8 respectively. ) 


Likert Scale | 1/2/)3/4/5]6/17]8 
Values 1/0, 0/1)1/)6/6)]0 


Table 5.7: Table : Likert scale of survey question-16 


Variance : 
N 


Variance = 0? = So (mi — p)?/N 


Standard Deviation : 


N 
StandardDeviation = 0 = So(m — p)?2/N 


a 


MOOC platforms | Mean( 1) | Variance(o”) | Standard deviation(c) 
Coursera 6.33 0.62 0.78 
Udemy 5.86 2.38 1.54 


Table 5.8: Table : Variance and Standard deviation of MOOC platforms 


From the table 5.8, Udemy has a high variance and standard deviation compared 
to Coursera. Based on the calculation of the user ratings, Udemy provides a better 
user experience than Coursera. 


Chapter 6 


Discussion 


In this section, the research questions are explained in detail. The first and second 
questions are well explained using the tasks and survey conducted. 


Research question 1: 


1. How effective, efficient and satisfactory are the two MOOC platforms (Udemy and 
Coursera)? 


Usability is determined by how well a characteristic adapts to the user’s needs and 
environment. When users first encounter the interface, they should navigate it easily 
to achieve their goals without relying on outside knowledge and knowing how to 
use it. It should be as user-friendly as possible and lead the user down the most 
accessible and least challenging path. Therefore, it must be familiar with the user’s 
environment, which requires considering their constraints, including the environment. 
Mainly the usability involves Navigation, Familiarity with the interface, consistency, 
error prevention, feedback to the user, visual clarity and Flexibility, and efficiency. 


A system or product depends entirely on the particular and unique situation in which 
it is used, the nature of the task, how the user will perform it, and the equipment used. 
The usability of a system can be measured using observable and quantifiable usability 
metrics to achieve these three goals: Effectiveness, Efficiency, and Satisfaction level. 


This research question results will be evaluated by taking the responses given by the 
users. The survey was conducted on fifteen users, and the results have been gathered. 


Fifteen users were given eight tasks to be completed to experiment. The tasks were 
explained to the users by the authors, and the author noted each task’s timings using 
a stopwatch on the phone while the user was performing the tasks on both MOOC 
platforms (Coursera and Udemy), respectively. The task timings of every participant 
for both MOOC platforms are tabulated and mentioned in Appendix A. The noted 
timing was converted into seconds(s). 


Note: Tasks done by the users are on their respective electronic devices (laptops 
and mobiles), and the processing power and other internet issues were ignored. 


The first task given to the user is to log in/ signup into the MOOC platforms, and 
the threshold value for the task1 for Coursera is 64.5s and for Udemy is 66.6s, where 
most of the users were able to complete the task within the threshold time. Few of 
them were not able to complete the task in time. Total successful tasks for task 1 in 
Coursera are 8 out of 15, and in Udemy, the successful tasks are 9 out of 15. 


The second task given to the user is to search the specified course(Machine Learning) 


Al 
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in MOOC platforms. The threshold value for Coursera is 22.2s and for Udemy is 
30.86s, where 12 out of 15 users could complete the task within the threshold time 
in Coursera, and in Udemy, 9 out of 15 completed the task in time. 


The third task is to read the course content/ course information of the Machine 
learning subject on both platforms, and the threshold for this task in Coursera is 
204.33s and in Udemy is 205.53s. 6 out of 15 users completed tasks in the time in 
Coursera and 9 out of 15 users completed tasks in time in Udemy. 


The fourth task given to the user is to enroll in the course and play the introduction 
videos for solving the quiz. Here the threshold value for Coursera is 14s and for 
Udemy is 18. 9 out of 15 users completed tasks in Coursera, and in Udemy, only 5 
out of 15 users were able to complete the tasks. 


The fifth task is to complete the quiz given to the users based on the introduction 
videos. 8 out of 15 users completed tasks successfully in both MOOC platforms, 
where the threshold value for this task for Coursera is 294.53s and for the Udemy is 
247.2s. 


The sixth task is to update the profile or upload the user’s picture and details on 
the respective platforms. The threshold time for task 6 in Coursera is 91.46s and for 
Udemy is 99.6s. In Coursera, 11 out of 15 users completed tasks within the threshold 
time, and in Udemy, 9 out of 15 users completed the task within the threshold time. 


The seventh task given to the user is to see the notifications regarding the course 
enrolled on the platforms. The threshold value for Coursera is 27.8s, and for Udemy, 
it is 23.33s. 7 out of 15 users completed the task in Udemy; only 4 out of 15 people 
completed the task within the threshold time in Coursera. 


The final task given to the users is to share the course on both platforms with their 
colleagues/ friends. The threshold value is 27.26s in Coursera, and Udemy is 31.73s. 
12 out of 15 people completed the task within the threshold time on both platforms. 


For calculating the Effectiveness, the above timing was taken into consideration. 
Effectiveness will be calculated by measuring the completion rate of the tasks given 
to the users by the total number of tasks undertaken. 


Number of tasks completed successfully 


Effectiveness = x 100 


Total Number of tasks undertaken 
(6.1) 


From the above formula, Effectiveness calculated for Coursera is 57.5% and for 
Udemy is 58.3%. Here Effectiveness of Udemy is better when compared to Coursera. 
Efficiency is calculated based on the successful tasks throughout the course, and it 
is measured by calculating the time taken by the user to complete the specific tasks. 


R N 
py aes 
j=ltl i=l 


Efficiency = — TR 


(6.2) 
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Where, 
N is the number of tasks, 


R is the number of users, 
nij is the result of task I by user j 
if the user completes the task, then nij=1 if not, nij=0 


tij is the time spent by the user j to complete a task. If the task is not completed 
successfully, the time is measured until the user quits the task. 


Efficiency of Coursera = 0.027goals/sec. (6.3) 


Efficiency of Udemy = 0.020goals/sec. (6.4) 


So from the above values, the Coursera and Udemy’s efficiency is negligible. Task 
Satisfaction level is calculated by taking the Likert scale values of each task com- 
pleted by the user. After completing the tasks, users were given a Likert scale table 
with an SEQ (Single ease questionnaire) for every task on both MOOC platforms. 


The values of each user’s satisfaction level with every task were taken. All the values 
were noted to calculate the task level satisfaction level and overall satisfaction level 
of the tasks given to the users. The repeated value of the Likert scale for all the tasks 
is taken. That value is multiplied by the number of repeated times and then divided 
by the number of users. It will give the satisfaction level of all tasks performed by 
every user; now, taking the average of each task satisfaction level gives the overall 
platform satisfaction level. All the calculations were mentioned in chapter 5. 


For every task, the satisfaction level of all the participants was calculated. When the 
satisfaction levels were compared, Coursera and Udemy were 5.63 (Fairly easy) and 
5.61 ( Fairly easy), respectively. So the result is that both the Task level satisfaction 
of Coursera and Udemy are similar. 


Research question 2: 


How does the user experience of two major MOOC platforms (Coursera and Udemy) 
compare to each other? 


This research question is explained by the results obtained from surveying the users. 
The results obtained from the users are evaluated statistically, to arrive at some 
conclusions. 


The survey is conducted among the same fifteen users. In addition, this section ex- 
plains the survey results in terms of relevant usability heuristics. The survey was 
conducted on the MOOC platforms Coursera and Udemy, and the users were given 
tasks to complete on the MOOC platforms. 


Users were given an overview of the tasks defined on the MOOC platform. The 
user had to complete eight tasks on the MOOC platform. The user first selected 
one MOOC platform to do the task and to complete the same task on another plat- 
form. During the performance of the task, the user’s task timings were recorded. 
After completing the tasks on the MOOC platform, the user was asked to complete 
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a questionnaire via a Google form provided by the author. Discussion about each 
survey question mentioned in chapter 4 is explained below. 


Survey question 1 was used to know the user’s familiarity with both MOOC plat- 
forms. By getting to know about the user’s familiarity, the author can explain the 
tasks to be performed by the user to help the user know about MOOC platforms. 


Survey question 2 was mapped with two usability heuristics: Visibility of system 
status and aesthetic and minimalist design. While registering into the MOOC plat- 
forms, password strength is shown when the password is entered. This type of feed- 
back comes under the Visibility of system status, and both the Udemy and Coursera 
have this feature. A website must display relevant information with a minimalistic 
design that the user can easily understand. Coursera and Udemy have required in- 
formation only while logging in or signing up. 


Survey question 3 was mapped to recognition rather than recall, consistency, and 
standard. Recognition and recall are defined as if the user wants to search for a 
course in the search bar; the user should not have to remember each word of the 
search he/she makes because Coursera and Udemy both have the functionality to 
remember the previously searched results. In this question, consistency and stan- 
dard mean the user should recognize the similar feature in the Coursera and Udemy 
compared to the standard features of other MOOC platforms, like having the same 
search option. 


Survey question 4 was mapped to match between the system and the real world. The 
user is asked to enroll in the specified course and watch the introduction videos. The 
interface of the MOOC platforms (Coursera and Udemy) should match between its 
system and the real world, as the introduction part should be separated from the 
rest of the videos. Udemy and Coursera follow the Match between the system and 
the real world by having the introduction videos while enrolling in the course. 


Survey question 5 was mapped to User control and freedom. Sometimes users may 
make mistakes and navigate to an unnecessary page and need to go back to the 
required page, where user control and freedom play a crucial role. A site should 
have clear marks on where the user is and where they can navigate by showing the 
selection in each menu. MOOC platforms (Udemy and Coursera) have this feature 
in their menus while playing any video. 


Survey question 6 was mapped to the Visibility of system status and error preven- 
tion. While taking a quiz, the important thing is that the site should give feedback 
to the user on whether the answer is saved and submitted. This Visibility of system 
status is observed in both MOOC platforms. Error prevention is another important 
aspect where the user needs to be intimated to answer the quiz question and submit 
it. In Coursera, the submit button appears when the user answers every question 
displayed. In contrast, Udemy users need to answer or skip each question until the 
final submission button appears. 


Survey question 7 was mapped to the Visibility of system status. The user task is to 
upload/update the profile picture of each MOOC platform. By the end of a task, the 
user needs to be notified about the update notification of the profile picture. There 
is no feedback in both MOOC platforms (Coursera and Udemy) as these platforms’ 
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profile picture is included in the profile section, where the user needs to save the 
whole profile section to save the picture. 


Survey question 8 was mapped to Match the system and the real world. Notifica- 
tions are present on both MOOC platforms (Coursera and Udemy). The icon on 
these platforms represents similar icons as the other social media applications like 
Facebook and LinkedIn. It is easier to recognize the icon, and all the notifications 
of the course offer and the current pursuing courses on the MOOC platforms are 
displayed. 


Survey question 9 was mapped to Flexibility and efficiency of use. Here the user’s 
task is to share the course on both MOOC platforms. Getting to know the user 
interaction while sharing the course allows the user efficient use and Flexibility of 
sharing the course with others like many other social networks (Whatsup, Facebook). 
It is easier to get acquainted with the functions of the interface so that the MOOC 
platform’s efficiency can be determined. 


Survey question 10 is mapped to help and documentation. Most of the time, users do 
not need any guidance or help for the tasks performed. However, documentation is 
needed if any problem is developed and the user needs guidance. It may be necessary 
to provide the help and documentation. It helps the users with easy navigation to 
focus on the task that needs to be done. 


Survey question 11 was mapped to help users recognize, diagnose and recover from 
errors. For any user, if any error occurs, an error message should appear so that the 
user can rectify the mistake and continue the tasks to be performed with the help of 
indicating the problem by suggesting a solution. 


Survey question 12, Survey question 13, and Survey question 14 were developed to 
understand user skills. It helps the authors know the user’s knowledge and computer 
skills while performing the tasks. 


respondents. 


Survey question 15 is the overall experience of the user on Coursera. Variance and 
standard deviation were calculated from the results collected from the users. 0.62 
and 0.78 are the Variance and standard deviation of Coursera, respectively. 


Survey question 16 is the overall experience of the user on Udemy. Variance and 
standard deviation were calculated by collecting results from the user. 2.38 and 1.54 
are the Variance and standard deviation of Coursera, respectively. 


Variance : 
N 


Variance = 0? = So (mi — p)’/N 
Where, 
m, is value of each task 
jt is mean value of all tasks 
N is total number of tasks 
Standard Deviation : 
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N 
Standard Deviation = 0 = Slim — p)?/N 


a 


Comparing the variance and standard deviation of the MOOC platforms, Udmey 
has more variance and standard deviation than the Coursera in terms of the users’ 
overall experience. 


Chapter 7 


Conclusions and Future Work 


7.1 Conclusions 


This study answers the question of which MOOC platform among Coursera and 
Udemy has a better user experience by using two different approaches. The first 
approach is to calculate both MOOC platforms’ effectiveness, efficiency, and satis- 
faction level (Coursera and Udemy). For this, specific tasks were given to the fifteen 
equally skilled people and asked to perform the tasks, and the time taken to finish 
each task by each user was noted. These timings are tabulated, and the threshold 
value is calculated. The threshold value is used to calculate each MOOC platform’s 
effectiveness and efficiency. 


A Single Easy Questionnaire (SEQ) method is used, where a Likert scale rating for 
each task by each individual is noted. The repeated values of the Likert scale are 
used to calculate the satisfaction level of each MOOC platform. 


The second approach calculated the variance and standard deviation using the sur- 
vey evaluation results. For this, users were given a questionnaire-based survey where 
each question is mapped against the heuristic principles to evaluate the user experi- 
ence of both MOOC platforms (Coursera and Udemy). The survey respondents are 
the same users who had performed each task to calculate the effectiveness, efficiency, 
and satisfaction level. By this approach, Udemy has more variance and standard 
deviation compared to the Coursera. 


This study concludes that Coursera and Udemy’s effectiveness, efficiency and satis- 
faction level are comparatively similar. However, from the survey results considering 
user experience, users have chosen Udemy over Coursera. 


7.1.1 Future Work 


For future work, this study can be explored in various ways like 


e One can also use HCI design principles to compare MOOC platforms. 


e Comparative usability evaluation can be performed on other MOOC platforms 
like Edx, Udacity, and Khan Acadamy. 


e This study can be extended to paid MOOC platforms like LinkedIn learning. 


e The survey can be conducted on large number of users. 


AT 
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Chapter 7. Conclusions and Future Work 


e This study can be extended to other evaluation methods like Interviews, Cog- 
nitive walkthrough, and pluralistic walkthrough. 
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Appendix A 


Appendix 


This section is about each participant’s time for finishing the tasks given to them in 
both Coursera and Udemy. NOTE: In task 4, time is recorded only for the enrollment 
of the course and time for playing the introduction videos was not included because 
for Coursera time for introduction videos is 33 min and in Udemy it is 25min. 


Tasks | Coursera | Udemy 
Task1 15s 67s 
Task2 38 Al 
Task3 108 141 
Task4 16 19 
Task5 257 166 
Task6 147 159 
Task7 oe 12 
Task8 76 88 


Table A.1: Time taken (in sec) by participant 1 for finishing each task 


Tasks | Coursera | Udemy 
Task1 18 63 
Task2 19 AO 
Task3 187 180 
Task4 10 15 
Task5 264 181 
Task6 107 62 
Task7 34 oT 
Task8 12 28 


Table A.2: Time taken (in sec) by participant 2 for finishing each task 
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Tasks | Coursera | Udemy 
Task1 14 10 
Task2 39 Al 
Task3 242 145 
Task4 15 Ze 
Task5 317 166 
Task6 171 209 
Task7 31 10 
Task8 121 80 


Table A.3: Time taken (in sec) by participant 3 for finishing each task 


Tasks | Coursera | Udemy 
Task1 60 64 
Task2 20 30 
Task3 187 242 
Task4 12 19 
Task5 344 340 
Task6 84 61 
Task7 32 45 
Task8 10 25 


Table A.4: Time taken (in sec) by participant 4 for finishing each task 


Tasks | Coursera | Udemy 
Task1 15 67 
Task2 38 Al 
Task3 108 141 
Task4 18 pal 
Task5 257 166 
Task6 147 159 
Task7 33 12 
Task8 76 88 


Table A.5: Time taken (in sec) by participant 5 for finishing each task 


Tasks | Coursera | Udemy 
Task1 81 73 
Task2 21 34 
Task3 247 226 
Task4 13 18 
Task5 256 351 
Task6 81 121 
Task7 3l 24 
Task8 20 23 


Table A.6: Time taken (in sec) by participant 6 for finishing each task 


Tasks | Coursera | Udemy 
Task1 62 56 
Task2 20 29 
Task3 243 302 
Task4 16 20 
Task5 345 289 
Task6 74 61 
Task7 31 25 
Task8 11 24 


Table A.7: Time taken (in sec) by participant 7 for finishing each task 


Tasks | Coursera | Udemy 
Task1 80 59 
Task2 20 59 
Task3 188 226 
Task4 20 22 
Task5 286 351 
Task6 72 59 
Task7 21 25 
Task8 10 20 


Table A.8: Time taken (in sec) by participant 8 for finishing each task 


Tasks | Coursera | Udemy 
Task1 62 56 
Task2 20 29 
Task3 248 302 
Task4 10 21 
Task5 345 289 
Task6 74 61 
Task7 31 25 
Task8 11 24 


Table A.9: Time taken (in sec) by participant 9 for finishing each task 


Tasks | Coursera | Udemy 
Task1 85 54 
Task2 21 30 
Task3 242 184 
Task4 13 18 
Task5 236 161 
Task6 71 94 
Task7 12 15 
Task8 10 15 


Table A.10: Time taken (in sec) by participant 10 for finishing each task 
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Tasks | Coursera | Udemy 
Task1 54 71 
Task2 20 23 
Task3 135 141 
Task4 14 19 
Task5 341 242 
Task6 81 144 
Task7 30 20 
Task8 10 9 


Table A.11: Time taken (in sec) by participant 11 for finishing each task 


Tasks | Coursera | Udemy 
Task1 116 102 
Task2 14 21 
Task3 243 186 
Task4 15 19 
Task5 301 292 
Task6 65 64 
Task7 31 24 
Task8 10 13 


Table A.12: Time taken (in sec) by participant 12 for finishing each task 


Tasks | Coursera | Udemy 
Task1 92 102 
Task2 21s 17 
Task3 244s 272 
Task4 16 10 
Task5 276 222 
Task6 65 116 
Task7 34 25 
Task8 10 2 


Table A.13: Time taken (in sec) by participant 13 for finishing each task 


Tasks | Coursera | Udemy 
Task1 102 100 
Task2 12 15 
Task3 222 194 
Task4 10 15 
Task5 291 221 
Task6 62 65 
Task7 21 15 
Task8 10 12 


Table A.14: Time taken (in sec) by participant 14 for finishing each task 
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Tasks 


Coursera 


Udemy 


Task1 


112 


65 


Task2 


10 


13 


Task3 


221 


201 


Task4 


T2 


14 


Task5 


302 


201 


Task6 


71 


59 


Task7 


12 


16 


Task8& 


12 
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Table A.15: Time taken (in sec) by participant 15 for finishing each task 
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Explore v What do you want to learn? 


New! Prepare for a management career in healthcare with a master’s degree from Northeastern University. 


Learn without 


limits 


Start, switch, or advance your career with more than 5,000 courses, 
Professional Certificates, and degrees from world-class universities 


and companies. 
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Figure A.1: Coursera User Interface 


Future-ready skills on your schedule | Learn why Udemy is trusted by 10.5K+ companies around the world x 


iF 
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udemy Categories (a Search for anything 


Learning that gets you 


Skills for your present (and your future). Get 
started with us. 


Abroad selection of courses 


Choose from 185,000 online video courses with new additions published every month 


Python 


Excel Web Development JavaScript DataScience AWS Certification Drawing 


Figure A.2: Udemy User Interface 
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